Method and device for monitoring the environment of a robot

ABSTRACT

A method for monitoring the environment of a robot including at least one iteration of a detection phase including the following steps:—acquisition, at a measurement instant, of an image of the depth of the environment, referred to as a measurement image, by the at least one 3D camera,—readjustment of the reference and measurement images, and—detection of a change relating to an object in the environment of the robot by comparing the reference and measurement images. A device that implements such a method and a robot equipped with such a device is also provided.

TECHNICAL FIELD

The present invention relates to a method for monitoring the environmentof a robot, with a view to detecting objects located in the environment.It also relates to a monitoring device implementing such a method, and arobot equipped with a monitoring device.

The field of the invention is, non-limitatively, that of the field ofrobotics, in particular the field of industrial robotics or of, forexample medical or domestic, service robots or also collaborativerobots, also called “cobots”.

STATE OF THE ART

The utilization of robots, such as for example robotized arms, in anon-enclosed environment in which humans/objects are likely to bemoving, requires the use of a functionality for detecting saidobjects/humans, in order to prevent collisions.

It is known to equip robots with sensors detecting objects in contactwith, or in immediate proximity to, the robot. These solutions have thedrawback that they do not allow detection of distant objects in order toanticipate collisions, which makes it necessary to restrict the speed ofmovement of the robot in order to limit the force of any impact to belowan acceptable threshold.

It is also known to use 2D or 3D sensors installed in the environment ofthe robot, so as to be able to monitor predefined regions around thisrobot with a view to detecting objects and anticipating collisions. Thissolution requires intervention in the work environment in order todefine the siting of the sensors and to install these sensors, which isa time-consuming, complex and costly operation. In addition, if theenvironment is changed, or the robot is moved, the intervention needs tobe repeated. Finally, depending on the movements of the robot, areas maybe obscured, which requires the utilization of several sensors,sometimes in complex configurations.

An aim of the present invention is to overcome the aforementioneddrawbacks.

A further aim of the present invention is to propose a method and adevice for monitoring the environment of a robot that is morecost-effective, less complex and less time-consuming to install and use,while providing detection of distant objects in order to anticipatecollisions.

A further aim of the present invention is to propose such a device andsuch a method that allows rapid implementation and easy adaptation tochanges in the environment of the robot.

A further aim of the present invention is to propose such a device andsuch a method making it possible to carry out a detection of objectswithin the vicinity of the robot with sufficient detection reliabilityto be able to be used as an anti-collision safety device.

A further aim of the present invention is to propose such a device andsuch a method making it possible to adapt the path of the robot as afunction of the movements in its environment.

A further aim of the present invention is to propose such a device andsuch a method making it possible to detect the obstacles at a longdistance from the robot in order to allow the latter to move at highspeed.

DISCLOSURE OF THE INVENTION

At least one of these aims is achieved with a method for monitoring theenvironment of a robot comprising:

-   -   a phase of obtaining a depth image of the environment of said        robot, called reference image, by at least one 3D camera carried        by said robot; and    -   at least one iteration of a detection phase comprising the        following steps:        -   acquiring, at an instant of measurement, a depth image of            said environment, called measurement image, using said at            least one 3D camera,        -   registering said reference and measurement images, and        -   detecting a change with respect to an object in the            environment of said robot by comparing said reference and            measurement images.

The method according to the invention proposes carrying out monitoringof the environment of a robot based on depth images obtained by 3Dcameras carried by the robot itself, and not by 3D cameras installed inthe environment of the robot. Thus, when the environment of the robotchanges, or when the robot is moved, no intervention is necessary toredefine, or alter, the siting of the 3D cameras. The method accordingto the invention is thus less complex, more cost-effective and lesstime-consuming to install and use.

In addition, the method according to the invention proposes carrying outmonitoring of the environment of a robot using 3D cameras, with abroader scope than contact sensors or proximity sensors such ascapacitive sensors. Thus, if objects are detected, the method accordingto the invention makes it possible to anticipate collisions without theneed to restrict the speed of movement of the robot.

By “object in the environment” or “object” is meant any fixed or mobileobject, living or not, that may be found in the environment of a robot.For example, this may be an object located in the environment of therobot, such as an operator or a person, a conveyor, a table or workarea, a trolley, etc.

By “robot” is meant a robot in any of its forms, such as a robotizedsystem, a mobile robot, a vehicle on wheels or tracks such as a trolleyequipped with an arm or a manipulator system, or a robot of thehumanoid, gynoid or android, type, optionally provided with movementmembers such as limbs, a robotized arm, etc. In particular, a robot canbe mobile when it is capable of moving or comprises moving parts.

In the context of the present invention, by “registering” two depthimages is meant determining the relative positioning of the two images,or the positioning of said images within a common frame of reference.The common frame of reference can be the frame of reference of one ofthe two images, or a frame of reference of the imaged scene.

A depth image can be an image obtained in the form of a point cloud or apixel matrix, with one item of distance or depth information for eachpixel. Of course, the point data can also comprise other items ofinformation, such as items of luminous intensity information, ingrayscale and/or in colour.

Generally, each image is represented by numerical data, for example datafor each point belonging to the point cloud. The processing of eachimage is carried out by processing the data representing said image.

By “3D camera” is meant any sensor capable of producing a depth image.Such a sensor can be for example a time-of-flight camera and/or astereoscopic camera, and/or a camera based on a structured lightprojection, which produces an image of its environment according to afield of view on a plurality of points or pixels, with one item ofdistance information per pixel.

A 3D camera can also comprise one or a plurality of, for example opticalor acoustic, distance point sensors, fixed or equipped with a scanningdevice, arranged so as to acquire a point cloud according to a field ofview.

A field of view can be defined as an area, an angular sector or a partof a real scene imaged by a 3D camera.

According to non-limitative embodiment examples, the change with respectto an object can be an appearance of said object in the environment ofthe robot, a disappearance of an object previously present in theenvironment of the robot, an alteration of the size or the position ofan object previously present in the environment of the robot, etc.

According to a first embodiment, the reference image can correspond to asingle depth image, acquired by the at least one 3D camera at an instantof acquisition.

In this case, it seems important to use a 3D camera with a field of viewthat is as wide as possible, with a view to capturing a depth image ofthe environment of the robot that is as complete as possible. In fact,the larger the reference image, the larger the environment monitored inproximity to the robot.

According to a second embodiment that is in no way limitative, thereference image can be constructed from several depth images.

In this case, the phase of obtaining the reference image can comprisethe following steps:

-   -   acquiring, sequentially, at least two depth images, at different        instants of acquisition and for different positions of the at        least one 3D camera; and    -   constructing the reference image from said sequential depth        images.

In other words, the reference image is constructed by combining, orconcatenating or else fusing, several depth images taken at differentinstants.

Above all, the fields imaged by the at least one 3D camera, at twoinstants of acquisition, are different because the position and/or theorientation of the at least one 3D camera is altered between these twoinstants of acquisition. Thus, it is possible to construct a largerreference image. It is even possible to construct a reference image ofthe entirety of the environment of the robot, by carrying out as manyacquisitions as necessary.

According to an embodiment example, at least one 3D camera can be mobileon the robot. Thus, the field of view of said 3D camera can be alteredwithout the need to move the robot.

Alternatively or in addition, at least one 3D camera can be fixed on therobot. In this case, the field of view of said 3D camera can be alteredby moving the robot, or a part of the robot on which the 3D camera isfixed. The robot, or the part of the robot, carrying the 3D camera canbe moved following a predetermined path, making it possible to image theentirety, or at least a large part, of the environment of the robot inseveral acquisitions.

According to an embodiment, when the sequential depth images compriseareas of overlap, then the construction of the reference image can becarried out by:

-   -   - detecting said areas of overlap, for example by comparing said        sequential depth images with one another; and p1 combining the        sequential images with one another, for example by concatenating        said depth images, as a function of said areas of overlap.

This combining can be performed in particular by implementing knownmethods of correlation search or error function minimization, betweenimages in the areas of overlap.

Alternatively, or in addition, the construction of the reference imagecan be carried out as a function of the configuration of the at leastone 3D camera, at each instant of acquisition.

This technique makes it possible to construct the reference image, evenwhen the sequential depth images do not contain areas of overlap.

The configuration of a 3D camera includes its position and/ororientation in space, for example in a frame of reference or acoordinate system associated with the scene or with the environment inwhich the robot is moving. Knowledge of this configuration of a 3Dcamera makes it possible to position its field of view in the scenecoordinate system.

According to an embodiment example, the position, at an instant ofacquisition, of the at least one 3D camera can be determined as afunction of a geometric configuration of said robot at said instant ofacquisition.

The geometry of the robot makes it possible to determine theconfiguration of a 3D camera fixed to the robot, in particular itsposition and its orientation, in a coordinate system associated with thescene.

In the particular case of a robotized arm comprising at least onerotationally mobile segment, the geometric configuration of the robot isgiven by the dimension of each segment and the angular position of eachsegment, this angular position being given by the motor or the jointmoving said segment. By taking account of these items of information oflength and of angular positions of the segments, it is possible todetermine the exact position and/or orientation of each 3D camera fixedto the robot.

According to an embodiment, the robot can be equipped with a single 3Dcamera.

In this case, each depth image at an instant corresponds to a depthimage taken by said 3D camera.

According to another embodiment, the robot can be equipped with several3D cameras, in particular with different fields of view.

In this embodiment, and in no way limitatively, each depth image at aninstant of acquisition can be a composite depth image constructed fromseveral depth images each acquired by a 3D camera at said given instant.

In this case, the step of acquiring a depth image at an instant ofacquisition can comprise the following operations:

-   -   acquiring, at said instant of acquisition, using at least two of        said 3D cameras, individual depth images; and    -   constructing a composite depth image from said individual depth        images.

Thus, at each instant of acquisition, it is possible to obtain acomposite depth image corresponding to a larger field of view, connected(in a single piece) or not. As a result, each composite depth imagerepresents the real scene according to a larger field of view comparedwith the embodiment using a single camera. This makes it possible tocarry out monitoring of a larger part, or even of the entirety, of theenvironment of the robot.

According to an embodiment, when the individual depth images compriseareas of overlap, then the construction of the composite depth image,from said individual depth images, can be carried out:

-   -   by detecting said areas of overlap, for example by comparing        said individual depth images with one another; and    -   by combining said individual depth images with one another, as a        function of said areas of overlap, for example by concatenating        or fusing said individual depth images.

This combining can be performed in particular by implementing knownmethods of correlation search or error function minimization, betweenimages in the areas of overlap.

Alternatively, or in addition, the construction of the composite depthimage, from the individual depth images, can be carried out as afunction of the relative configurations of the 3D cameras with respectto one another.

This technique makes it possible to construct the composite depth image,even when the fields of view of the 3D cameras do not contain areas ofoverlap, so that the individual depth images acquired at a given instantby the 3D cameras do not contain areas of overlap.

The relative configuration of a 3D camera, with respect to another 3Dcamera, can comprise its relative position, and/or its relativeorientation, with respect to the relative position, or relativeorientation respectively, of the other 3D camera.

When the robot is equipped with one or more 3D cameras with differentfields of view, the detection phase can be carried out individually forat least one 3D camera, adopting as measurement image an individualdepth image taken by said 3D camera at the instant of measurement.

In this case, the individual depth image of the 3D camera is registeredwith the reference image, then compared with the reference image with aview to identifying, in the measurement image, a change with respect toan object in the individual depth image.

In particular, the detection phase can be carried out for eachindividual depth image of each 3D camera, if necessary.

The detection phase can then be carried out at the same time or in turnfor each 3D camera.

According to another embodiment, when the robot is equipped with several3D cameras with different fields of view, the measurement image at aninstant of measurement can be a composite depth image constructed fromseveral individual depth images acquired by several 3D cameras at saidinstant of measurement.

In this case, the step of acquiring a measurement image, at an instantof measurement, can comprise the following operations:

-   -   acquiring, at the instant of measurement, using several 3D        cameras, individual depth images; and    -   constructing a composite measurement image from said individual        depth images.

Thus, at each instant of measurement, it is possible to obtain acomposite measurement image corresponding to a larger field of view. Asa result, each composite measurement image represents the environment ofthe robot according to a larger field of view compared with theembodiment using a single 3D camera. This makes it possible to carry outmonitoring of a larger part, or even of the entirety, of the environmentof the robot.

According to an embodiment example, when the individual depth imagescomprise areas of overlap, then the construction of the compositemeasurement image, from said individual depth images, can be carriedout:

-   -   by detecting said areas of overlap, for example by comparing        said individual depth images with one another; and    -   by combining said individual depth images with one another, as a        function of said areas of overlap, for example by concatenating        or fusing said individual depth images.

Alternatively, or in addition, the construction of the compositemeasurement image, from the individual depth images, can be carried outas a function of the relative configurations of the 3D cameras withrespect to one another.

This technique makes it possible to construct the composite measurementimage, even when the fields of view of the 3D cameras do not containareas of overlap, so that the individual depth images acquired by the 3Dcameras, at the instant of measurement, do not contain areas of overlap.

The relative configuration of a 3D camera, with respect to another 3Dcamera, can comprise its relative position, and/or its relativeorientation, with respect to the relative position, or relativeorientation respectively, of the other 3D camera.

According to an advantageous characteristic, the detection of a changewith respect to an object can be carried out by utilizing the items ofdistance information from the measurement image.

In particular, this detection can be performed by detecting thatdistances measured in at least one area of the measurement image aredifferent from distances in the corresponding area or areas of thereference image.

Of course, the detection of a change with respect to an object can alsobe carried out by utilizing other available items of information, suchas for example an intensity, a grayscale and/or a colour, alone or incombination.

Furthermore, the type of object or at least its shape can also bedetermined, in particular from the distance measurements.

Of course, a movement or withdrawal of an object can also be detected inthe same way, by comparing the measurement and reference images.

According to an advantageous characteristic, when a change with respectto an object is detected in the measurement image, the detection phasecan also comprise a step of determining a distance relative to saidobject, by analysing said measurement image.

As a result, when an object is identified in an area of the measurementimage (by detecting distance differences or other criteria), therelative distance of the object is determined as being, for example, thedistance of the closest point belonging to said object in themeasurement image. Alternatively, the relative distance of the objectcan correspond to the average distance of the points belonging to saidobject in the measurement image.

According to another particularly advantageous characteristic, themethod according to the invention can comprise a step of triggering acommand of said robot, if a change with respect to an object is detectedin the measurement image.

Such a triggering can also be a function of a characteristic of theobject: the relative distance of the object, the type of object, thesize of the object, etc.

Examples of commands can comprise:

-   -   an emergency stop, in particular if an approaching object is        identified as an operator;    -   a change of speed or a deceleration;    -   an alteration of path, with for example generation of a bypass        path;    -   a triggering of a specific task, or a parameterization of a        task, for example if a trolley with objects to be handled is        detected.

It is also possible to define areas in the reference image that will besubject to specific processing if for example an object is detected inthese areas:

-   -   near/distant area;    -   area where an operator causes deceleration /exclusion area with        stopping of the robot;    -   area associated with the performance of a task (expected        position of a conveyor, etc.).

For example, a command of the robot can be triggered only when therelative distance of the object is below, or equal to, a predetermineddistance threshold.

In addition, the type of command can be a function of the relativedistance of the object. For example, when the relative distance of theobject is:

-   -   below or equal to a first threshold, a first command of the        robot can be triggered, such as for example a deceleration of        the robot;    -   below or equal to a second threshold, a second command of the        robot can be triggered, such as for example an alteration of its        movement path; and    -   below or equal to a third threshold, a third command of the        robot can be triggered, such as for example an emergency stop of        the robot.

Generally, in the present application, each depth image is acquired inthe form of a point cloud. In other words, when a 3D camera takes animage, it supplies a point cloud representing the depth image. Eachpoint of the point cloud is represented by the coordinates of saidpoint, or by an item of distance data and a solid angle, in a frame ofreference associated with the 3D camera at the moment of acquiring theimage.

According to embodiments of the invention, the different operationsapplied to the depth images can be performed directly on the pointclouds constituting said depth images. For example, the reference imagecan be produced by fusing point cloud images together. Similarly, acomparison or registration of a first depth image with respect to asecond depth image can be carried out by comparing or registering thepoint cloud representing the first depth image with the point cloudrepresenting the second depth image. However, this embodiment requiresnot insignificant resources and calculation time, which increase withthe number of points in each point cloud.

According to other embodiments of the invention, the point cloudrepresenting a depth image can be processed or segmented beforehand inorder to deduce therefrom a set of simple geometric shapes, such asplanes, so as to reduce the quantity of information to be processed.This prior processing can be carried out using known techniques, forexample by using algorithms of the RANSAC type. This prior processingthen supplies an image represented or modelled by a dataset indicatingthe type of geometric shapes thus detected, their dimensions, theirposition and their orientation. The functions applied to a depth image,such as registration and optionally comparison or detection of a change,can then be applied, not to the initial point cloud representing thedepth image, but to the dataset supplied by the prior processing. Thisprior treatment can be applied at any time during the performance of themethod according to the invention, such as for example:

-   -   directly to each depth image acquired by a 3D camera: in this        case no step of the method is applied to the point cloud;    -   to each composite depth image obtained, at an instant of        acquisition, for all of the 3D cameras: in this case, the        composite depth image is obtained by processing the point clouds        representing the individual depth images;    -   to the reference image or to the images used to construct the        reference image;    -   to the measurement image.

Thus, according to embodiments, the registration of the reference andmeasurement depth images can be carried out by analysing point cloudimages.

It can in particular be carried out by similarity, or correlation,search or minimization of distances between point clouds, or moregenerally using all the techniques currently known.

In this case, the comparison step for detecting a change can also becarried out on the point clouds.

According to other embodiments, the registration of the reference andmeasurement depth images can be carried out by analysing images modelledbeforehand in the form of geometric shapes.

These geometric shapes can in particular comprise planes.

The registration of the images can in particular be carried out bysimilarity, or correlation, search or minimization of distances betweengeometric shapes, or more generally using all the techniques currentlyknown.

In this case, the comparison step for detecting a change can also becarried out:

-   -   between the reference image and a measurement image modelled        beforehand in the form of geometric shapes; or    -   between the reference image modelled beforehand in the form of        geometric shapes, and a measurement image in the form of a point        cloud.

The registration of the reference and measurement depth images can alsobe carried out with a method for registering two depth images of a realscene comprising the following steps:

-   -   for each of said depth images:        -   detecting a plurality of several geometric shapes in said            depth image, and        -   determining at least one geometric relationship between at            least two geometric shapes of said plurality of geometric            shapes;    -   identifying geometric shapes common to said two images by        comparing the geometric relationships detected for one of the        images with those detected for the other of said images;    -   calculating, as a function of said common geometric shapes, a        geometric transformation between said images; and    -   registering one of said images, with respect to the other of        said images, as a function of said geometric transformation.

The geometric transformation can be in particular a linear or rigidtransformation. It can in particular be determined in the form of adisplacement matrix, representative for example of a translation and/ora rotation.

Thus, the registration method according to the invention proposesregistering with one another, or locating with respect to one another,two depth images of one and the same real scene, not by using theobjects present in the scene, but using geometric relationships betweengeometric shapes identified in each depth image.

The identification, in each image, of the geometric shapes and geometricrelationships between these shapes requires fewer calculation resourcesand less calculation time, than the identification of real objects ofthe scene in each image.

In addition, it is simpler and quicker to compare the geometricrelationships with one another than to compare the real objects of thescene with one another. In fact, the geometric shapes and theirrelationships are represented by a much smaller quantity of data to beprocessed than the quantity of data representing the real objects of thescene.

The geometric shapes can be geometric elements capable of beingdescribed or modelled by equations or equation systems, such as forexample: planes, lines, cylinders, cubes, etc.

According to embodiments, the registration method of the invention cancomprise detecting geometric shapes that are all of the same type (forexample planes alone).

According to further embodiments, the registration method of theinvention can comprise detecting geometric shapes of different typesfrom among a finite set (for example planes and cubes).

Advantageously, the geometric relationships used can be invariant by thegeometric transformation sought.

Thus, for example, angles and distances are invariant by the geometrictransformations of the rotation and translation type.

For each image, the detection step can comprise detecting at least onegroup of geometric shapes all having a similar orientation or one andthe same orientation with respect to a predetermined reference directionin the scene.

Geometric shapes can be considered as having a similar orientation orone and the same orientation with respect to a reference direction whenthey are all oriented according to a particular angle with respect tothis reference direction within a predetermined range of angulartolerances, for example of +/−5 degrees, or +/−10 degrees. Thisparticular angle can be for example 0 degrees (parallel orientation) or90 degrees (perpendicular orientation). It is well understood that thegeometric shapes of a group can furthermore have a non-parallelorientation to one another.

The detection of a group of geometric shapes can comprise identifying,or classifying in this group, geometric shapes in a set of geometricshapes detected beforehand in the depth image, or detecting theseparticular geometric shapes in the depth image.

In the case where a shape is a line, the orientation of the shape cancorrespond to the orientation of said line with respect to the referencedirection. In the case where a shape is a two-dimensional plane, theorientation of the shape can correspond to the orientation of saidplane, or of its normal vector, with respect to the reference direction.Finally, when a shape is a three-dimensional shape (such as a cylinderor a cube), its orientation can be given by its main, or extension,direction, or an axis of symmetry.

The geometric shapes and their orientation can be determined by theknown techniques.

Generally, the point cloud is segmented or grouped in the form of areasor sets corresponding to, or capable of being modelled or approximatedby, one or more geometric shapes of a predetermined type. Then thedescriptive parameters of these geometric shapes are calculated by errorminimization methods such as the least squares method, by optimizing forexample parameters of a geometric equation in order to minimizedifferences with respect to the point cloud. The orientation of thegeometric shapes can then be deduced for example from the parameters ofthe equations that describe them.

Preferably, for each image, the detection step can comprise detecting:

-   -   a first group of geometric shapes, all having a first        orientation with respect to the reference direction, in        particular an orientation parallel to the reference direction;        and    -   at least one second group of geometric shapes, all having one        and the same second orientation with respect to the reference        direction, different from the first orientation, in particular        orthogonal to said first orientation.

Thus, the method according to the invention makes it possible to obtaintwo groups of shapes with different, and in particular perpendicular,orientations with respect to the reference direction. The geometricrelationships are determined between the shapes belonging to one and thesame group.

The geometric relationship sought between the shapes of a group can beidentical to, or different from, the geometric relationship soughtbetween the shapes of another group. For example, for one of the groupsthe geometric relationship sought between the shapes can be a distancerelationship, and for the other of the groups the geometric relationshipsought between the groups can be an angular relationship.

In a particularly preferred embodiment, the reference direction can bethe direction of the gravity vector in the scene.

The direction of the gravity vector can correspond to the orientation ofthe gravitational force.

In this embodiment, the detection step can comprise detecting a group ofgeometric shapes having a horizontal orientation in the scene, such asfor example shapes corresponding to real horizontal objects in thescene. Such horizontal geometric shapes can correspond to the floor, theceiling, a table, etc.

According to an advantageous, but in no way limitative, characteristic,the geometric relationship between two horizontal geometric shapes cancomprise, or be, a distance between said two shapes, in the direction ofthe gravity vector.

In other words, the distance sought between two horizontal shapes is thedistance separating said shapes in the vertical direction.

Still in the embodiment where the reference direction is the directionof the gravity vector in the scene, the detection step can comprisedetecting a group of geometric shapes having a vertical orientation inthe scene.

Such vertical geometric shapes can correspond to vertical objects in thescene, such as walls, doors, windows, furniture, etc.

According to an advantageous, but in no way limitative, characteristic,the geometric relationship between two vertical geometric shapes cancomprise at least one angle between the two geometric shapes.

In particular, the geometric relationship between two vertical shapescan comprise an angle between said shapes in the horizontal plane.

According to a preferred, but in no way limitative, embodiment, thedetection step can carry out a detection:

-   -   of a first group of horizontal geometric shapes, and    -   of a second group of vertical geometric shapes; with respect to        the gravity vector.

In this embodiment, the determination step determines:

-   -   one or more angle(s), measured in the horizontal plane, between        the vertical geometric shapes; and    -   one or more distance(s), measured in the vertical direction,        between the horizontal geometric shapes.

The reference direction can be represented by a reference vector, whichcan be any vector determined beforehand and indicating a direction and asense in the real scene.

In a particular case, the reference direction can be the direction ofthe gravity vector in the scene, or in other words the reference vectorcan be the gravity vector. This vector can then be used, in each image,to determine if a shape of said image has a specific orientation, forexample a vertical orientation or a horizontal orientation.

According to an embodiment, the reference vector, and in particular thegravity vector, can be detected and reported by a sensor for each image.Such a sensor can be for example an accelerometer.

According to another embodiment, the reference vector, and in particularthe gravity vector, can be determined, in each image, by analysing saidimage.

For example, in the case where the reference vector is the gravityvector, then each image can be analysed to detect a plane correspondingto the floor or to the ceiling, in the case of an interior scene: thegravity vector then corresponds to the vector perpendicular to thisplane. Generally, the floor or the ceiling are the largest planes in adepth image.

According to another embodiment example, when the depth image comprisesa colour component, then the colour component can be used to detect apredetermined plane, and this plane can be used to obtain the referencevector, and in particular the gravity vector.

Advantageously, the step of determining geometric relationships cancomprise, for each geometric shape, determining a geometric relationshipbetween said geometric shape and each of the other geometric shapes, sothat a geometric relationship is determined for each combination usingpairs of the geometric shapes.

When the detection step comprises detecting one or more groups ofgeometric shapes, the step of determining geometric relationships cancomprise, for each geometric shape of a group, determining a geometricrelationship between said geometric shape and each of the othergeometric shapes of said group, so that a geometric relationship isdetermined for each combination using pairs of the geometric shapes ofsaid group.

Thus, for a group comprising “n” geometric shapes, Σ₁ ^(n−1)k pairs ofgeometric shapes are obtained, and thus as many geometric relationships.

At least one geometric shape can be a line, a plane or athree-dimensional geometric shape.

In particular, if necessary, all the geometric shapes of one and thesame group, and more generally of all the groups, can be of the sametype.

In a particularly preferred embodiment, all the geometric shapes can beplanes.

In this case, in each image, the detection step can carry out anapproximation, by planes, of surfaces of the real scene appearing insaid image. Thus, an object having several faces is approximated byseveral planes, without the need to detect the entire object.

In addition, the geometric relationships are simpler to determinebetween planes.

In each image, the detection of the planes can be carried out in asimple manner using known algorithms, such as for example the RANSACalgorithm.

According to an embodiment example, the detection of planes in a depthimage can be carried out using the following steps:

-   -   for each point of the depth image, or the image as point cloud,        normals (N) of each point of the cloud are calculated by using,        for example, the depth gradient. The latter is obtained, in        practice, for each point, by subtracting the depth of the lower        point from that of the upper point (vertical gradient) and the        depth of the left point from that of the right point (horizontal        gradient). The normal at the point is then given by the vector        product of the vertical gradient vector and the horizontal        gradient vector.    -   at least one iteration of a step of calculating planes in the        point cloud by using the normals of the points in the point        cloud. This step comprises the following operations:    -   taking 3 points at random from the point cloud,    -   calculating the parameters of a plane passing through these 3        points, then    -   calculating the number of points of P belonging to this plane        according to criteria such as distances below a threshold or        near normals.

A plane can be considered as identified if a minimum number of pointshas been identified as belonging to it. The points belonging to a planeare removed from the set P so as not to be used for identifying thefollowing planes. The above step of calculating planes can be reiteratedas many times as desired.

The descriptive parameters of the plane can then be determined from thepoints identified as belonging to this plane, by calculating for examplethe parameters of an equation of this plane in the least-squares sense.

The step of detecting planes thus supplies a list of the planesidentified with their descriptive parameters and the set of pointsbelonging to them.

The set of planes can be processed within a single group.

Alternatively, among all the planes detected, the vertical planes can begrouped in a first group and the horizontal planes can be grouped withina second group. The geometric relationships can then be determinedbetween the planes of one and the same group, as described above.

According to an embodiment, the calculation of the geometrictransformation can comprise calculating a transformation matrixconstructed from:

-   -   a difference, at least in a given direction, between the        position of at least one geometric shape in one of said images        and the position of said at least one geometric shape in the        other of said images; and    -   a difference in orientation between at least one orthonormal        coordinate system associated with at least one geometric shape        in one of said images and said at least one orthonormal        coordinate system associated with said at least one geometric        shape in the other of said images.

According to an embodiment in which all the geometric shapes are planes,in particular a group of horizontal planes and a group of verticalplanes, the calculation of the geometric transformation can comprisecalculating a transformation matrix constructed from:

-   -   a distance, in the vertical direction, between the position of a        horizontal plane in one of said images and the position of said        plane in the other of said images;    -   two distances, in horizontal directions, between the respective        positions of two vertical planes that are not parallel or        orthogonal to one another in one of said images and the        respective positions of said planes in the other of said images;        and    -   a difference in orientation between an orthonormal coordinate        system associated with a vertical plane in one of said images,        and an orthonormal plane associated with said vertical plane in        the other of said images.

The distances in the horizontal directions can also be determined fromrespective positions, in the two images, of the straight lines ofintersection of two vertical planes that are not parallel or orthogonalto one another.

The transformation matrix thus obtained is complete and makes itpossible to fully register two depth images with one another.

According to another aspect of the invention, a device for monitoringthe environment of a robot is proposed, comprising:

-   -   at least one 3D camera, and    -   at least one calculation means;        configured to implement all the steps of the method for        monitoring the environment of a robot according to the        invention.

The calculation means can be a calculator, a processor, amicrocontroller, an electronic chip or any electronic component.

According to yet another aspect of the present invention, a robotequipped with a monitoring device according to the invention isproposed.

The robot according to the invention can be a robot in any of its forms,such as a robotized system, a mobile robot, a vehicle on wheels ortracks such as a trolley equipped with an arm or a manipulator system,or a robot of the humanoid, gynoid or android, type, optionally providedwith movement members such as limbs, a robotized arm, etc. Inparticular, a robot can be mobile when it is capable of moving orcomprises moving parts.

In particular, the robot according to the invention can comprise:

-   -   at least one mobile segment, and    -   several 3D cameras distributed around one of the mobile        segments.

DESCRIPTION OF THE FIGURES AND EMBODIMENTS

Other advantages and characteristics will become apparent on examiningthe detailed description of examples that are in no way limitative, andfrom the attached drawings, in which:

FIGS. 1a and 1b are diagrammatic representations of a non-limitativeembodiment example of a robot according to the invention;

FIG. 2 is a diagrammatic representation of a non-limitative embodimentexample of a method according to the invention;

FIG. 3 is a representation of an embodiment example of a method forregistering depth images, capable of being implemented in the presentinvention, and

FIG. 4 is a non-limitative and very simplified application example ofthe method in FIG. 3.

It is well understood that the embodiments that will be describedhereinafter are in no way limitative. Variants of the invention can beenvisaged in particular comprising only a selection of thecharacteristics described hereinafter, in isolation from the othercharacteristics described, if this selection of characteristics issufficient to confer a technical advantage or to differentiate theinvention with respect to the state of the prior art. This selectioncomprises at least one, preferably functional, characteristic withoutstructural details, or with only a part of the structural details ifthis part alone is sufficient to confer a technical advantage or todifferentiate the invention with respect to the state of the prior art.

In particular, all the variants and all the embodiments described can becombined with one another, if there is no objection to this combinationfrom a technical point of view.

In the figures, elements common to several figures retain the samereference sign.

FIGS. 1a and 1b are diagrammatic representations of a non-limitativeembodiment example of a robot according to the invention.

FIG. 1a is a diagrammatic representation of the robot in a profile view,and FIG. 1b is a representation in a front view of the distal segment ofthe robot.

The robot 100 in FIG. 1 is a robotized arm comprising several segments102-108: segment 102 being the base segment of the robot 100, andsegment 108 being the distal segment of the robot. Segments 104, 106 and108 are rotationally mobile by virtue of the joints 110-114, and motors(not shown) at these joints 110-114.

The distal segment 108 can be equipped with a tool, such as a gripper116 for example, as shown in FIG. 1 a.

The base segment can be fixed on a floor 118. Alternatively, the basesegment can be equipped with means making it possible for the robot tomove, such as for example at least one wheel or track.

According to the invention, the robot has at least one 3D camera onboard.

In the example of FIGS. 1a and 1 b, the robot 100 is equipped withseveral, specifically eight, 3D cameras 120.

Each 3D camera 120 can be a time-of-flight camera for example.

The 3D cameras 120 are arranged around a segment of the robot, inparticular around the distal segment 108.

The cameras 120 are more particularly distributed according to aconstant angular pitch.

Each 3D camera 120 makes it possible to produce a depth image of a partof the real scene constituted by the environment of the robot 100,according to a field of view 122, radial with respect to the distalsegment 108. The field of view 122 of a 3D camera 120 is different fromthe field of view 122 of another 3D camera.

According to an embodiment example, the fields of view 122 of twoadjacent 3D cameras contain an area of overlap beyond a certaindistance. Of course, it is possible for the fields of view 122 of twoadjacent 3D cameras not to contain an area of overlap.

As shown in FIG. 1 b, the set of fields of view 122 of the 3D cameras120 cover a full circular view around the distal segment 108. It isclear that an only partial view of the environment is detected for oneposition of the robotized arm 100. However, when the robotized arm ismoved, other parts of the scene will be viewed and detected by the 3Dcameras.

Of course, in other embodiment examples, the robot 100 can comprise 3Dcameras arranged differently on a segment, and/or arranged on differentsegments. In other configuration examples, the 3D cameras can bearranged so that their combined total field of view makes it possible tocapture the real scene around the robot in its entirety, at least forone position of the robot 100.

Moreover, the robotized arm 100 is equipped with a processing unit 124,which can be a computer, a calculator, a processor or similar. Theprocessing unit 124 is linked by wire or wirelessly to each of thecameras 120. It receives, from each 3D camera, each depth image acquiredby said 3D camera, in order to process said image. The processing unit124 comprises computer instructions in order to implement the methodaccording to the invention.

In the example shown, the processing unit 124 is shown as a separateindividual module. Of course, the processing unit 124 can be combinedwith, or integrated in, another module, or in a calculator of therobotized arm 100.

FIG. 2 is a diagrammatic representation of a non-limitative embodimentexample of a method for monitoring the environment of a robot, accordingto the invention.

The method 200, shown in FIG. 2, can in particular be implemented by therobot 100 in FIGS. 1a and 1 b.

The method 200 comprises a phase 202 of obtaining a depth image of theenvironment of the robot, without the presence of operators orunexpected objects. This image of the environment of the robot will beused as reference image in order to detect any change with respect to anobject in the environment of the robot.

The phase 202 comprises a step 204 of obtaining a depth image, at aninstant of acquisition, for a given configuration of the robotized arm.When the robot is equipped with a single 3D camera, the depth imagecorresponds to the image supplied by said single 3D camera.

When the robot is equipped with several 3D cameras, such as for examplethe robot 100 in FIGS. 1a and 1 b, then a composite depth image isconstructed from the individual depth images taken by said 3D cameras,at said instant of acquisition. In this case, step 204 comprises thefollowing steps:

-   -   during a step 206 carried out at said instant of acquisition,        each 3D camera takes an individual depth image; and    -   during a step 208, the individual depth images are combined, in        order to obtain a composite depth image for all of the 3D        cameras.

The combination of the individual depth images in order to obtain asingle composite depth image, at an instant of acquisition, can becarried out according to different techniques.

According to a first technique, when the individual depth imagescomprise areas of overlap, i.e. when the fields of view of the 3Dcameras contain areas of overlap, then the combination of the individualdepth images can be carried out by detecting these areas of overlap, andby using these areas of overlap to concatenate the individual depthimages with a view to obtaining a composite depth image.

According to a second technique, which can be used alone or incombination with the first technique, the combination of the individualdepth images can be carried out by using the relative configurations ofthe 3D cameras. In fact, the position and the orientation of each 3Dcamera is known, with the proviso, of course, that it is positioned in aknown manner on the robot. Consequently, by using the relative positionsand relative orientations of the 3D cameras with respect to one another,it is possible to position, with respect to one another, the individualdepth images taken by these 3D cameras. Specifically, for eachindividual depth image taken by a 3D camera, the position of the 3Dcamera corresponds to the centre or to the point of origin of saidindividual depth image, and the orientation of each 3D cameracorresponds to the direction in which the individual depth image wastaken. By using these two items of information, the individual depthimages can be positioned with respect to one another, in order to obtaina single composite depth image for all of the 3D cameras, at an instantof acquisition.

Step 204 can be reiterated as many times as desired, sequentially, atdifferent instants of acquisition, each instant of acquisitioncorresponding to a different configuration of the robot. Thus, for eachconfiguration of the robot, a composite depth image is obtained.

In particular, step 204 can be reiterated sequentially while the robotis moved, continuously or not, following a predetermined path, with aview to imaging the environment of the robot to a large extent, and inparticular in its entirety. Each iteration of step 204 makes it possibleto obtain a composite depth image.

During a step 210, the reference image is constructed from the differentcomposite depth images obtained sequentially for differentconfigurations of the robot. The construction of a reference image fromseveral composite depth images acquired sequentially at differentinstants of acquisition can be carried out according to differenttechniques.

According to a first technique, the sequential composite depth imagescan be acquired, making sure that they comprise areas of overlap. Inthis case, the construction of the reference image can be carried out bydetecting the areas of overlap between the composite depth images andusing these areas of overlap to concatenate the sequential compositedepth images with one another.

According to a second technique, which can be used alone or incombination with the first technique, the construction of the referenceimage from the sequential composite depth images can be carried out byusing the geometric configuration of the robot, for each depth image. Inthe case of a robotized arm, the geometric configuration of the robot isgiven by:

-   -   the dimensions of the different mobile segments of the robot:        these dimensions are known; and    -   the relative orientations of the mobile segments: these        orientations can be known from the joints, or from the motors        placed in the joints.

Thus, with knowledge of the geometric configuration of the robot at aninstant of acquisition, it is possible to position, within a coordinatesystem associated with the environment, the depth image obtained forthis instant of acquisition.

The reference image thus obtained is stored during a step 212. Thisreference image is thus constituted by the set of depth images acquiredand fused so as to constitute an image representing all or part of theenvironment of the robot.

In the example described, step 208 of constructing a composite depthimage from several individual depth images is carried out immediatelyafter the acquisition of said individual depth images. Alternatively,this step 208 can be carried out just before step 210 of constructingthe reference image. According to yet another alternative, steps 208 and210 can be carried out at the same time, within a single step alone,taking account of all the individual depth images acquired for each ofthe sequential instants of acquisition.

The method 200 also comprises at least one iteration of a detectionphase 220 carried out when the robot is in the process of operating.This detection phase 220 is carried out in order to detect, at aninstant of measurement, a change with respect to an object located inthe environment of the robot.

The detection phase 220 comprises a step 222 of acquiring a depth image,called measurement image, at the instant of measurement. Thismeasurement image will then be compared with the reference image, storedin step 212, in order to detect a change with respect to an objectlocated in the environment of the robot.

The measurement image, acquired at an instant of measurement, can be anindividual depth image acquired by a 3D camera. In this case, thedetection phase can be carried out individually for each individualdepth image acquired by each 3D camera, at said instant of measurement.

Alternatively, the measurement image acquired at an instant ofmeasurement can be a composite measurement image constructed from theindividual depth images acquired by all the 3D cameras at said instantof measurement. In this case, step 222 comprises a step 224 of acquiringan individual depth image using each 3D camera. Then, in a step 226, thecomposite measurement image is constructed from individual depth imagesacquired by all the 3D cameras, for example by using one of thetechniques described above with reference to step 208.

During a step 228, the measurement image, or the composite measurementimage, is registered with the reference image. The aim of thisregistration operation is to locate or position the measurement image inthe coordinate system or the frame of reference of the reference image.

The registration of the reference and measurement images can be carriedout using known techniques, such as techniques of registration bysimilarity search, or by correlation, or by minimization of distances.

The registration of the reference and measurement images can also becarried out using a view registration method, a non-limitativeembodiment example of which is described below with reference to FIGS. 3and 4.

Once the (composite) measurement image is registered with the referenceimage, the registered images are compared with one another, during astep 230, in order to detect a change with respect to an object in thecomposite measurement image.

In the embodiment presented, the comparison of the images is carried outfrom the distance measurements. It can comprise for example detectingareas of the measurement image with distances or positions differentfrom those in the corresponding areas in the reference image. Thisdifference can be due for example to the appearance, the disappearanceor the movement of an object or of an operator.

As explained above, this comparison can be carried out between images inpoint cloud form.

When the reference image is modelled by geometric elements, thiscomparison can be carried out either with a measurement image in pointcloud form or with a measurement image also modelled by geometricelements.

Choice of the comparison method can depend on the objects or theelements sought.

For example, in a situation where the environment of the robot isconstituted by a room with walls and items of furniture (conveyor, rackor cupboard, etc.) and where it is sought to detect objects ofindeterminate shape (operator, etc.), it can be advantageous:

-   -   to model the reference image with geometric elements, and in        particular planes;    -   to model the measurement image with geometric elements, and in        particular planes, for the registration operation; and    -   to use the measurement image in point cloud form in order to        detect the changes with respect to the objects.

Thus, the registration can be performed in a manner that is accurate andeconomical in terms of computing power, and the comparison operationmakes it possible to extract the measurement points corresponding to thedifferent objects, without assumptions as to their shape. It is thuspossible to analyse these objects, for example with a view toidentifying them.

When no significant difference is observed between the reference imageand the composite measurement image, then this iteration of thedetection phase 220 is terminated. A fresh iteration of the detectionphase 220 can be carried out at any time.

When a significant difference with respect to an object is detectedbetween the reference image and the composite measurement image, thisdifference is analysed in order to decide if an action is necessary. Inthis case, a command of the robot is triggered during a step 234.

For example, if an object appears, the detection phase can also comprisea step 232 of calculating a relative distance of said object. Thisrelative distance of said object is given in the (composite) measurementimage, since the latter comprises an item of distance information foreach pixel of said image.

When the distance determined in step 232 is below a predeterminedthreshold, then a command of the robot is triggered during step 234.Examples of commands can comprise:

-   -   an emergency stop, in particular if an approaching object is        identified as an operator;    -   a change of speed or a deceleration;    -   an alteration of path, with for example, generation of a bypass        path;    -   a triggering of a specific task, or a parameterization of a        task, for example if a trolley with objects to be handled is        detected.

It is also possible to define areas in the reference scene that will besubject to a specific processing if for example an element is detectedin these areas. The following are for example to be defined:

-   -   a near area, and/or a distant area;    -   a safety area where an operator causes deceleration, or an        exclusion area with stopping of the robot;    -   an area associated with the performance of a task (expected        position of a conveyor, etc.).

With reference to FIGS. 3 and 4, an embodiment example of a method forregistering depth images, capable of being implemented in the presentinvention, and in particular during step 228, will now be described.

FIG. 3 is a diagrammatic representation of a non-limitative embodimentexample of a method for registering depth images according to theinvention, using planes as geometric shapes.

The method 300, shown in FIG. 3, makes it possible to register with oneanother, or to locate with respect to one another, a first depth imageof a scene (such as a reference image) and a second depth image of thesame scene (such as a measurement image), in a common frame ofreference, which can be that of one of the two images or a frame ofreference associated with said scene.

The method 300 comprises a phase 302 ₁ of processing the first depthimage.

The processing phase 302 ₁ comprises a step 304 ₁ of detecting planes inthe first image. This step 304 ₁ can be carried out using knowntechniques, such as for example a technique using the RANSAC algorithm.According to an embodiment example that is in no way limitative, thestep 304 ₁ of detecting planes can be carried out as follows,considering that the first depth image is represented by a point clouddenoted P1. A first step calculates the normals (N) of each point of thepoint cloud P, using for example the depth gradient: this is obtained,in practice, for each point, by subtracting the depth of the lower pointfrom that of the upper point (vertical gradient) and the depth of theleft point from that of the right point (horizontal gradient). Thenormal to the point is then given by the vector product of the twogradient vectors. Then a second step uses the normals of the points forcalculating the planes in the point cloud. This step consists of:

-   -   taking 3 points at random from the point cloud P1,    -   calculating the parameters of a plane passing through these 3        points, then    -   calculating the number of points of P1 belonging to this plane        according to criteria such as distances below a threshold or        near normals.

Step 304 ₁ of detecting planes can be reiterated a predetermined numberof times. A plane can be considered as identified if a minimum number ofpoints has been identified as belonging to it. The points belonging to aplane are removed from the set P1 of points, in order not to be used foridentifying the following planes.

Step 304 ₁ of detecting planes supplies a list of the planes identifiedwith their descriptive parameters and the set of points of P belongingto them.

It should be noted that this method described for the step of detectingthe planes is also applicable to the modelling of depth images in theform of geometric elements as described above.

Then, an optional step 3061 makes it possible to detect the gravityvector in the first image. This gravity vector corresponds to the vectorhaving the same direction as the gravitational force, expressed in thecoordinate system of the 3D camera that took the image. In order toobtain this vector, an approximately horizontal plane that is asextensive as possible is sought. This plane is considered as beingperfectly orthogonal to the gravitational force. The normal to thisplane gives the gravity vector. Alternatively, the gravity vector can bereported by a sensor, such as an accelerometer or an inclinometer,detecting said gravity vector at the moment the first depth image istaken.

Then, a step 308 ₁ makes it possible to select, from among all theplanes identified in step 304 ₁, a first group of horizontal planes.Each plane having a normal parallel to the gravity vector (orsubstantially parallel with an angular tolerance, for example of +/−10degrees) is considered as being horizontal. Step 308 ₁ thus supplies afirst group of horizontal planes.

During a step 310 ₁, for each horizontal plane of the first group, ageometric relationship is detected between this horizontal plane andeach of the other horizontal planes forming part of the first group. Inparticular, in the embodiment presented, the geometric relationship usedbetween two horizontal planes is the distance between these planes inthe direction of the gravity vector, i.e. in the vertical direction. Thegeometric relationships identified in step 310 ₁ are stored.

A step 312 ₁ makes it possible to select, from among all the planesidentified in step 304 ₁, a second group of vertical planes. Each planehaving a normal perpendicular to the gravity vector (or substantiallyperpendicular with an angular tolerance, for example of +/−10 degrees)is considered as being vertical. Step 312 ₁ thus supplies a second groupof vertical planes.

During a step 314 ₁, for each vertical plane of the second group, ageometric relationship is detected between this vertical plane and eachof the other vertical planes forming part of the second group. Inparticular, in the embodiment presented, the geometric relationship usedbetween two vertical planes is the relative angle between these planes.The geometric relationships identified in step 314 ₁ are stored.

A processing phase 302 ₂ is applied to the second depth image, at thesame time as the processing phase 302 ₁ or after the processing phase302 ₁. This processing phase 302 ₂ is identical to the processing phase302 ₁, and comprises steps 304 ₂-314 ₂ respectively identical to steps304 ₁-314 ₁.

During a step 316, each geometric relationship between the horizontalplanes, identified for the first image in step 310 ₁, is compared witheach geometric relationship between the horizontal planes, identifiedfor the second image in step 310 ₂. When two geometric relationshipscorrespond, then this indicates that these geometric relationshipsconcern the same horizontal planes on the two images. Thus, thehorizontal planes common to the two images are identified.

During a step 318, each geometric relationship between the verticalplanes, identified for the first image in step 314 ₁, is compared witheach geometric relationship between the vertical planes, identified forthe second image in step 314 ₂. When two geometric relationshipscorrespond, then this indicates that these geometric relationshipsconcern the same vertical planes on the two images. Thus, the verticalplanes common to the two images are identified.

At the end of these steps, a mapping of the respective vertical andhorizontal planes of the two images is obtained.

The method according to the invention can also comprise additional stepsof validating the mapping of the planes of the two images. It is thus inparticular possible to verify:

-   -   if the normal vectors and the distances to the origin of the        planes are similar;    -   if the extents of the planes, represented by their convex        envelopes, are superposed;    -   if the points forming the convex envelope of a plane identified        on the first image are close to the surface of the plane        identified on the second image;    -   if their colour distributions or their appearance, for example        obtained from the respective colour histograms of the two        planes, are similar.

These additional validation steps can be performed, for example, bycalculating comparison heuristics of the planes two by two, if necessaryby applying a geometric transformation (for example as described below)in order to express the planes of the second image in the coordinatesystem of the first image and thus make them comparable.

During a step 320, a geometric transformation, in the form of ahomogeneous displacement matrix, is calculated by considering theposition and orientation of the common planes identified in each of theimages. This matrix makes it possible for example to express the planesof the second image in the coordinate system of the first image. It thusmakes it possible to determine the displacement or the difference inposition, in the scene, of the sensor or of the 3D camera that made itpossible to acquire each of the images.

In the embodiment presented, the rotation of one of the images, withrespect to the other of the images, is determined using common verticalplanes. In fact, the gravity vector, associated with a vertical planenormal vector, being orthogonal to gravity, gives an orthonormal basis.The two orthonormal bases that correspond to one another in two viewsdirectly give the angle of rotation of the sensor on each of the axes.The angles of rotation on the three axes are thus calculated for eachcorresponding plane and averaged. Then, the horizontal translationvector is calculated by mapping the two straight lines of intersectionbetween two orthogonal vertical planes. At this stage, it is alsopossible to obtain the vertical translation by defining the quadraticerror matrix associated with the two planes in order to obtain thevector that minimizes this matrix.

In addition, the common horizontal planes make it possible to calculatethe vertical translation. By calculating the difference in distance tothe origin of each plane in the two images, it is possible to find atranslation vector oriented according to the normal of the horizontalplanes.

The displacement matrix determined in step 320 can then be applied toone of the images in order to register it with the other of the images,during a step 322.

The two images are then registered with one another in one and the samecoordinate system or, in other words, both positioned in one and thesame coordinate system.

The images thus registered can then be utilized independently of oneanother.

They can also be fused with one another or in a broader 3Drepresentation, according to known techniques. This fusion can inparticular be carried out between point clouds, or between identifiedgeometric shapes.

FIG. 4 is a diagrammatic representation of a, very simplified,application example of a method for registering depth images accordingto the invention, and in particular of the method 300 in FIG. 3.

FIG. 4 shows two depth images 402 and 404 of one and the same scene (forexample a reference image and a measurement image) according to twodifferent fields of view. The coordinate system (X, Y, Z) is acoordinate system associated with the field of view of the image 402, orwith the associated sensor. It is therefore different from a coordinatesystem associated with the field of view of the image 404.

Each of the images 402 and 404 is processed in order to detect, in eachimage, vertical planes and the horizontal planes. The result obtainedfor the image 402 is given by the plane base image 406, and that for theimage 404 is given by the plane base image 408. Thus, for each image,horizontal planes and vertical planes are identified. The normal vectorof each plane is also indicated.

In the first image 402, the horizontal planes detected in step 308 ₁ arethe following:

-   -   a horizontal plane h1 of which:        -   the normal vector h1 is the vector Nh1=(0, 1, 0)        -   the centre of gravity is the point Ch1=(1, −1, 2)    -   a horizontal plane h2 of which:        -   the normal vector is the vector Nh2=(0, −1, 0)        -   the centre of gravity is the point Ch2=(−2, 2, 1).

The coordinates can be for example in metres.

The distance relationship between these two horizontal planes, detectedin step 310 ₁ is given by a projection of the points Ch1 and Ch2 (or ofthe corresponding vectors from the origin of the coordinate system) ontoone of the normal vectors, for example Nh1. It is given by the followingrelationship:

distance(Ch1, Ch2)=abs(Nh1·Ch1−Nh1·Ch2)=3.

where “abs” is the absolute value and “·” is the scalar product.

Still in the first image 402, the vertical planes detected in step 312 ₁are the following:

-   -   a vertical plane v1 of which:        -   the normal vector is the vector Nv1=(1, 0, 0)        -   the centre of gravity, defined as the average of the            positions of all the 3D points belonging to the plane v1, is            the point Cv1=(−2, 1, 1)    -   a vertical plane v2 of which:        -   the normal vector is the vector Nv2=(0, 0, −1)        -   the centre of gravity is the point Cv2=(1, 1, 3).

The angular relationship between these two vertical planes, detected instep 314 ₁, is given by the following relationship: angle(Nv1, Nv2)=90°.

In the second image 404, the horizontal planes detected in step 308 ₂are the following:

-   -   a horizontal plane h′1 of which:        -   the normal vector h1 is the vector Nh′1=(−0.150, 0.985,            0.087)        -   the centre of gravity is the point Ch′1=(2.203, −0.311,            1.203)    -   a horizontal plane h′2 of which:        -   the normal vector is the vector Nh′2=(0.150, −0.985, −0.087)        -   the centre of gravity is the point Ch′2=(−1.306, 2.122,            2.074).

The distance relationship between these two horizontal planes, detectedin step 310 ₂, and calculated as above, is given by the followingrelationship:

distance(Ch′1, Ch′2)=abs(Nh′1·Ch′1−Nh′1·Ch′2)=3.

Still in the second image 404, the vertical planes detected in step 312₂ are the following:

-   -   a vertical plane v′1 of which:        -   the normal vector is the vector Nv′1=(0.853, 0.174, −0.492)        -   the centre of gravity is the point Cv′1=(−1.156, 1.137,            1.988)    -   a vertical plane v′2 of which:        -   the normal vector is the vector Nv′2=(−0.5, 0, −0.866)        -   the centre of gravity is the point Cv′2=(2.402, 1.658,            2.242).

The angular relationship between these two vertical planes, detected instep 314 ₂, is given by the following relationship: angle(Nv′1,Nv′2)=90°.

By comparing the angular relationships angle(Nv1, Nv2) and angle(Nv′1,Nv′2), an equality is detected:

angle(Nv 1, Nv 2) = angle(Nv^(′)1, Nv^(′)2).

This makes it possible to confirm that the vertical planes (v1, v2) inthe first image 402 are indeed the vertical planes (v′1, v′2) in thesecond image 404.

In addition, by comparing the relationships distance(Ch1, Ch2) anddistance(Ch′1, Ch′2), an equality is detected:

distance(Ch 1, Ch 2) = distance(Ch^(′)1, Ch^(′)2).

This makes it possible to confirm that the horizontal planes (h1, h2) inthe first image 402 are indeed the horizontal planes (h′1, h′2) in thesecond image 404.

By using the characteristics of the vertical planes and of thehorizontal planes common to the two images 402 and 404, a homogeneousdisplacement matrix is calculated. In the example given, the homogeneousdisplacement matrix (R, T) is the following:

Rotation  R = (0^(^(∘)), 30^(^(∘)), 0^(^(∘)))Translation  T = (0.20, 0.50, 0.05).

It should be noted that there is a relationship between the differentparameters defining the planes and the R and T vectors. For example:Nv′1=R×Nv1 and Cv′1=R×Cv1+T.

The rotation R is calculated using the vertical planes v1, v2, v′1, v′2and the gravity vector.

The horizontal translation components (in x and z) of T are calculatedusing the vertical planes v1, v2, v′1 and v′2.

The vertical translation component (y) of T is calculated using thehorizontal planes h1, h2, h′1 and h′2.

The transformation T thus calculated is applied to the second image 404so as to express this image 404 in the coordinate system (X, Y, Z) ofthe first image 402.

Of course, the invention is not limited to the examples that have justbeen described, and numerous modifications may be made to these exampleswithout exceeding the scope of the invention.

In the examples described, the robot comprises several 3D cameras. Ofcourse, the number of 3D cameras is non-limitative and the robot cancomprise one or more cameras. When the robot comprises a single camera,steps 208 and 226 of constructing composite depth images are not carriedout.

1. A method for monitoring the environment of a robot comprising: a phase of obtaining a depth image of the environment of said robot, called reference image, by at least one 3D camera carried by said robot; and at least one iteration of a detection phase comprising the following steps: acquiring, at an instant of measurement, a depth image of said environment, called measurement image, by said at least one 3D camera; registering said reference and measurement images; and detecting a change with respect to an object in the environment of said robot by comparing said reference and measurement images.
 2. The method according to claim 1, characterized in that the phase of obtaining the reference image comprises the following steps: acquiring, sequentially, at least two depth images, at different instants of acquisition and for different positions of the at least one 3D camera; and constructing the reference image from said sequential depth images.
 3. The method according to claim 2, characterized in that the construction of the reference image is carried out as a function of the configuration of the at least one 3D camera, at each instant of acquisition.
 4. The method according to claim 3, characterized in that the position, at an instant of acquisition, of the at least one 3D camera is determined as a function of a geometric configuration of the robot at said instant of acquisition.
 5. The method according to claim 1, characterized in that, when the robot is equipped with several 3D cameras with different fields of view, the step of acquiring a depth image, at an instant of acquisition, comprises the following operations: acquiring, at said instant of acquisition, using at least two of said 3D cameras, individual depth images; and constructing a composite depth image from said individual depth images.
 6. The method according to claim 5, characterized in that the construction of the composite depth image, from the individual depth images, is carried out as a function of the relative configurations of the 3D cameras with respect to one another.
 7. The method according to claim 1, characterized in that the detection phase is carried out individually for at least one 3D camera, by adopting as measurement image an individual depth image taken by said 3D camera at the instant of measurement.
 8. The method according to claim 1, characterized in that, when the robot is equipped with several 3D cameras with different fields of view, the step of acquiring a measurement image comprises the following operations: acquiring, at the instant of measurement, using several 3D cameras, individual depth images; and constructing a composite measurement image from said individual depth images.
 9. The method according to claim 8, characterized in that the construction of the composite measurement image is carried out as a function of the relative positions of the 3D cameras with respect to one another.
 10. The method according to claim 1, characterized in that the detection of a change with respect to an object in the environment of the robot is carried out by utilizing items of distance information from the measurement image.
 11. The method according to claim 1, characterized in that it comprises a step of triggering a command of the robot, if a change with respect to an object is detected in the measurement image.
 12. The method according to claim 1, characterized in that the registration of the reference and measurement depth images is carried out by analysing point cloud images.
 13. The method according to claim 1, characterized in that the registration of the reference and measurement depth images is carried out by analysis of images modelled beforehand in the form of geometric shapes.
 14. The method according to claim 13, characterized in that the registration of the reference and measurement depth images comprises the following steps: for each of said images: detecting a plurality of geometric shapes in said image; and determining at least one geometric relationship between at least two geometric shapes of said plurality of geometric shapes; identifying geometric shapes common to said two images by comparing the geometric relationships detected for one of the images with those detected for the other of the images; calculating, as a function of said common geometric shapes, a geometric transformation between said images; and registering one of said images, with respect to the other of said images, as a function of said geometric transformation.
 15. A device for monitoring the environment of a robot comprising: at least one 3D camera; and at least one calculation means; configured to implement all the steps of the method for monitoring the environment of a robot according to claim
 1. 16. A robot equipped with a monitoring device according to claim
 15. 17. The robot according to claim 16, characterized in that it comprises: at least one mobile segment, and several 3D cameras distributed around one of the mobile segments. 